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STATEMENT OF FOCUS 



The Wisconsin Research and Development Center for Cognitive Learning 
focuses on contributing to a better understanding of cognitive learning by 
children and youth and to the improvement of related educational practices. 

The strategy for research and development is comprehensive. It includes 
basic research to generate new knowledge about the conditions and processes 
of learning and about the processes of instruction, and the subsequent develop- 
ment of research-based instructional materials, many of which are designed for 
use by teachers and others for use by students. These materials are tested and 
refined in school settings. Throughout these operations behavioral scientists, 
curriculum experts, academic s.cholars, and school people interact, insuring 
that the results of Center activities are based soundly on knowledge of subject 
matter and cognitive learning and that they are applied to the improvement of 
educational practice. 

This Technical Report is from the Individually Guided Instruction in English 
Language, Composition, and Literature Project in Program 2. General objectives 
of the Program are to establish rationale and strategy for developing instructional 
systems, to identify sequences of concepts and cognitive skills, to develop 
assessment procedures for those concepts and skills, to identify or -velop in- 
structional materials associated with the concepts and cognitive skills, and to 
generate new knowledge about instructional procedures. Contributing to these 
Program objectives, the long-range objective of the English Project is to install 
and test materials for individually guided instruction in language, composition, 
and literature. Prerequisite activities include formulating behavioral objectives 
for students and teachers, based on a content and concepts outline, and devel- 
oping measurement instruments related to the behavioral objectives. 
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ABSTRACT 



The purpose of this study Vs^as to estimate the reliability of various 
measures of writing behavior using 500~word samples instead of 1000 words. 
The themes of 135 Eighth Graders were collected over a period of 6 weeks, 
one them® per week. The topics, expository and narrative, were selected 
Jointly by teachers and students as part of the normal curriculum in English. 
Samples of at least 1000 words were obtained from each student and the 
criteria of Hunt (1965) were\ used to determine which sentences and T-units 
to include for tabulation. Two 500-word divisions of the 1000-word samples 
were made. The themes written during the first 3 v^^eeks were compared to 
those of the second 3 weeks, and the odd pages were compared to the even 
pages . 

Eight of the tabulated structures (clauses of comparison, deferred sub- 
ject, cleft sentence, "the more, the merrier" construction, subjunctive word 
order, special which clauses, adjective complements, and adverb comple- 
ments) occurred very infrequently and were essentially uncorrelated (£< .26) 
from sample to sample. Adjective, adverb, and noun clauses were more fre- 
quent and correlated .35, .31, and .08, respectively, in the first-half /second- 
half samples, and .46, .42, and .26 in the odd/even pages samples. The 
five synopsis scores (clause length, clauses per T— unit, T— unit length, 

T-units per sentence, and sentence length) correlated .48, .22, .56, .48, 
and .62 between first half and second half respectively, and . 69, .54, .74, 
.65, and .77 between odd-page and even-page samples. 
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THE PROBLEM 



Hunt (1965) used 1000-word writing samples 
from which he tabulated eleven syntactical 
structures and five synopsis scores. The work 
required to tabulate such a large amount of 
writing is arduous, and if a smaller sample can 
be used which will still give an accurate meas- 
ure of writing behavior, much menial labor can 
be spared. The purpose of the present study 
was to provide some data on the reliability of 
the Hunt measures when 500 words of 1000- 
word samples are used. 

Some very old studies have made inferences 
about what length a writing sample should be. 
Frogner (1933)- thought that for some measures 
one needed to analyze 50,000 to 60,000 words 
before an accurate measure of group perform- 
ance was achieved. She made no estimate of 
the sample size required from an individual. 
This high number of words was especially 
needed when tabulating errors and specific 
categories of clauses . Other measures , such 
as the proportions of simple, compound, and 
complex sentences, produced by groups seemed 
to stabilize after 10,000 words had been tabu- 
lated. Frogner also noted that the kind of writ- 
ing influenced the complexity of the sentences . 
She showed that expository writing produced a 
higher percentage of complex sentences than 
narrative, which in turn produced a higher pro- 
portion of complex sentences than those pro- 
duced in letter-writing. 

Anderson (1937) tested the feasibility of 
using samples from individuals as small as 150 
words to tabulate sentence length. He con- 
cluded that 150-word samples were too small 
to reliably measure even the most objective 
indices. He postulated that sentence length 
would vary with the situation and the subject 
matter . 

Chotlos (1944) made a systematic compari- 
son of the reliability of type-token ratios (num- 
ber of unique words divided by the number of 
words) obtained from 100-, 500-, 1000-, and 
1500-word samples written by individuals. The 
type-token ratios for two 100-word samples 



correlated .38, while two 500-word samples 
correlated .67, 1000-word .81, and 1500- 
word .83. Though these correlations showed 
a clear increase with sample size, there was 
some confounding involved since, as sample 
size increased, the number of words between 
samples decreased. For example, the 1500- 
word samples were the first and second half 
of a 3000-word corpus, but the 1000 words 
were the first and last thirds of this same cor- 
pus . Thus, 1000 words of writing had taken 
place between the two 1000-word samples, but 
none had occurred between the 1500-word sam- 
ples . Whatever effect writing has on writing, 
it became greater with the smaller samples, 
until with 100 words a massive 2800 words had 
intervened between the two samples. The type- 
token ratio is also critically dependent upon 
the number of words in a way that other meas- 
ures (sentence length, clause length, T-unit 
length) are not. The types of words are accumu- 
lated rapidly as one begins tabulating. As 
tabulating continues the rate at which new 
types of words are added declines, but word 
tokens are jdded at a constant rate throughout. 

Recent studies have settled upon 1000 words 
per S as a suitable body of writing for purposes 
of analysis (Hunt, 19 65; Blount, Fredrick, & 
Johnson, 1968). One can ask, however, 
whether a smaller sample will result in meas- 
ures that approach the reliability of the same 
measures obtained from a larger sample,, If 
a 500-word sample can replace a 1000--word 
sample with only a slight decrease in relia- 
bility, the savings of time and effort are ob- 
vious . 

Taking under consideration the usual level 
of variability in measures of writing, it was 
decided that whenever a variable accounted 
for one-fourth of the variance in two succes- 
sive samples ( £ > .5) that variable was of some 
value as a measure of group performance. 
Whenever one-half of the variance was 
identifiable (r_> .7) the measure was of value 
for assessing individual performance. 
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II 

EXPERIMENTAL PROCEDURE 



The S.S were Eighth Graders in a large public 
junior high school in Wisconsin. High and low 
ability S.s were drawn from the Eighth Grade 
population (242 students in the classrooms of 
the participating teachers) . Among these were 
72 females and 63 males. Seventy-two ^s had 
above average scores (119-145) on the Cali- 
fornia Test of Mental Maturity and 63 were be- 
low average (90-106). 

During the first part of the 19 66-67 school 
year each ^ wrote one in-class theme per week 
as part of the normal work in his English class. 
£s did not know that the themes were being col- 
lected from the teachers, copied, and returned 
by the experimenters. The topics of the themes 
were the usual descriptive, narrative, and ex- 
pository assignments decided upon jointly by 
the students and teachers. The successive 
weekly themes were collected until each of the 
135 Ss had produced at least 1000 words of 
text. Generally this was accomplished by 
about the sixth theme. 

From the copies the themes were typed triple- 
spaced in preparation for the tabulation of vari- 
ous measures of writing ability. The guidelines 
developed by Hunt (1965) were used in tabulat- 
ing 17 measures, which included the following 
eleven subordinate structures, the total amount 
of subordination, and five synopsis scores: 

1. Adjective clauses: clause structures 
which functioned as adjectives in 
modifying a noun or pronoun. 

<That was the place where he lost 
hi's keys .> 

2. Adverb clauses: clause structures 
which functioned as movable adverbs, 

<I heard noises when I woke up.> 

3. Noun clauses: clause structures which 
occurred in the nominal positions of 
subject, direct object, object of prep- 
osition, and appositive. 

< That there are good teenagers is 
never mentioned. > 



4. Clauses of comparison: adverb-like 
clauses which were not usually mov- 
able and which used comparatives 
such as so that , more than , as long 
as, etc. 

<The clouds seemed so low that you 
could catch one.> 

5. Deferred subject: clause structures 
which were the "logical" subjects of 
the sentence but which followed the 
"grammatical" subject 

< It was good that such a thing hap- 
pened .> 

6. Cleft sentence: clause structures 
(other than "logical" subjects) caused 
by it as the "grammatical" subject. 

<It's the fog that keeps the fumes from 
escaping .> 

7. "The more, the merrier" : sentences 
which involved a the more this , the 
more that statem,ent . 

<The more he worried, the more he 
felt sorry for himself. > 

8. Subjunctive: use of different word 
order to express the subjunctive mood. 

< I would have been killed had it not 
been for him.> 

9. Special which clauses: which clauses 
that modified an entire sentence. 

<When we come in the morning we can 
go to the gym, which I think is nice.> 

10. Adjective complement: clauses that 
complemented an adjective and could 
not be moved. 

<She was sure that I had left home.> 

11. Adverb complement: clauses that fol- 
lowed verbs like seem, appeared , 
feel , etc. 

<She looked as if she had seen a ghost. > 

12. All subordinate clauses: the total of 
1-11 above. 
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For the synopsis scores, the number of words, 
clauses, T-units, and sentences were tabu- 
lated according to these guidelines; Words 
were all letter sequences that, regardless of 
spelling, could be considered a unit of speech. 
Contractions were counted as two words, and 
proper nouns (Yellowstone National Park) were 
counted as one word. Clauses were all main 
and subordinate structures that contained a 
subject and a finite verb phrase. T-units were 
defined as "one main clause plus the subordi- 
nate clauses attached to or embedded within 
it [Hunt, 1965, p. 49]." Sentences were the 
words written between a capital letter and some 
end punctuation. As an example of each of 
these terms, note 'the following sentence which 
contains three T-units marked by double slashes, 
a main clause within each T-unit, three subor- 
dinate clauses marked by a single slash and a 
symbol, and 42 words; < If I had a million dol- 
lars (adv) / I think / (N) that I would do a lot 
of things, // but I would put most of the money 
in the First National Bank // and I would start 
a job / (adv) so people wouldn't call me a lazy 
man. //> The tabulations of words , clauses, 
T-units, and sentences were then used to com- 
pute these synopsis scores; 

13. Clause length: words per clause. 

14. Clauses per T-unit. 

15. T-unit length; words per T-unit. 



16. T-units per sentence. 

17. Sentence length; words per sentence. 

Under Hunt's guidelines all sentence frag- 
ments, garbles, direct discourse, and ques- 
tions were deleted from consideration. The 
17 measures were tabulated independently by 
tv^ro raters who had been trained previously 
and had used the tabulating system in previous 
work for the experimenters . In past work the 
correlations between the two raters hau ranged 
from .84 to 1.00 on the various indices tabu- 
lated. All discrepancies between the two raters 
were resolved by the experimenter (SJ) who had 
originally trained the raters . A detailed dis- 
cussion of each measure can be found in Blount,, 
Fredrick, and Johnson (1968). 

When typed sentence-by-sentence and triple- 
spaced, each ^'s writing sample became a docu- 
ment of about 12 pages. Two comparisons be- 
tween halves of the 1000-word samples were 
then made. The first comparison involved split- 
ting the sample at the sentence or T-unit nearest 
the 500-word midpoint. Each of the 17 scores 
obtained from the first half was correlated with 
the respective score from the second half. The 
second comparison was between the odd pages 
of the typed sample and the even pages. Each 
of the 17 scores from the odd pages (to the end 
of the sentence or T-unit nearest 500 words) 
was correlated with the respective score from 
the even pages . 






RESULTS AND DISCUSSION 



The means and standard deviations from 
135 odd-page samples from the odd-even pages 
split are presented in Table 1 . The equivalent 
tables for even-page samples and first-half/ 
second-half samples are not presented since 
all four tables were practically identical. The 
means presented include eleven tabulations of 
subordinate structures , the total of all eleven 
kinds of subordinate structures, and five syn- 
opsis scores. In 500 words of Eighth-Grade 
writing, an average of 22 subordinate struc- 
tures appeared; roughly 5 of these were adjec- 
tive clauses, 9 adverb clauses, and 7 noun 



clauses. Each clause averaged about eight 
words in length, and appeared with a frequency 
of three clauses per every two T-units. T- 
units themselves averaged 13 words long, and 
one-third of the time there was more than one 
T-unit in a sentence. Sentence length averaged 
more than 1 6 words . Each of these statistics 
closely approximated the results found in pre- 
vious studies (Hunt, 1965; Blount, et al . . 
1968). [Adjustments were made to frequency 
counts so that 500- and 1000-word samnles 
could be compared.] 



Table 1 

Means and Standard Deviations Tabulated from 
the 500-Word Writing Samples of 135 Eighth Graders 



Variable 


All 135 Ss 
Mean sd 


63 Males 
Mean sd 


7 2 Females 
Mean sd 


7 2 High IQ 
Mean sd 


63 Low 10 
Mean sd 


Adjective Clauses 


5.3 


2.9 


5.4 


3.1 


5.2 


2.7 


5.5 


3.0 


5.1 


2.8 


Adverb Clauses 


8.8 


4.2 


8.3 


4.2 


9.3 


4. 2 


8.4 


4.2 


9.3 


4.3 


Noun Clauses 


7.0 


3.8 


6.7 


4.0 


7.2 


3.6 


6 . 8 


3.8 


7.2 


3.8 


Clauses of Comparison 


.4 


.7 


.5 


.7 


.3 


.7 


.4 


.7 


.4 


.7 


Deferred Subject 


.1 


.3 


.0 


. 2 


.1 


.4 


.1 


,3 


.1 


.4 


Cleft Sentence 


.1 


.3 


.1 


.4 


.1 


.3 


.1 


.3 


’ .1 


.4 


"The More, the Merrier" 


.0 


.1 


.0 


. 2 


.0 


— 


.0 


. 2 


.0 




Subjunctive 


.0 


. 2 


.0 


.3 


.0 




.0 




.0 


.3 


Special Which Clauses 


.1 


. 2 


.1 


. 2 


.0 


. 2 


.1 


. 2 


.0 


. 2 


Adjective Complement 


.1 


.4 


.1 


. 3 


. 2 


.5 


.1 


.4- 


.1 


.5 


Adverb Complement 


.1 


. 2 


.0 


.2 


.1 


.3 


.1 


. 2 


.0 


.2 


All Subordinate Clauses 


22.0 


6.2 


21.3 


6 . 6 


22.6 


5.9 


21 .6 


6.6 


22.5 


5.7 


Clause Length 


8.2 


1 . 2 


8.4 


1.3 


8.1 


1.1 


8.3 


1.3 


8.2 


1.1 


Clauses per T-unit 


1.6 


. 2 


1.6 


. 2 


1.6 


. 2 


1.6 


.2 


1.6 


.2 


T-unit Length 


13.1 


2.7 


13.2 


2.7 


13.0 


2.7 


13.1 


2.9 


13.1 


2.5 


T-units per Sentence 


1.3 


.2 


1.3 


. 2 


1.2 


. 2 


1.2 


. 2 


1.3 


. 2 


Sentence Length 


16.4 


3.5 


16.9 


3.7 


16.0 


3.3 


16.0 


3.1 


16.9 


3.9 
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To compare the differf^nces between odd/ 
even page samples, _t tests were computed on 
all 17 dependent measures. These tests were 
not significant nor were there significant dif- 
ferences in comparisons of the first-half/second' 
half samples. There were also no significant 
differences when the mean scores of the 63 
males were compared to the means of the 7 2 
females and neither were any significant dif- 
ferences observed in the comparison of high 
and low IQ ^s . 

The feasibility of using 500-word samplet, 
to obtain measures of subordination and writing 
maturity can be seen more clearly in the corre- 
lation coefficients presented in Table 2. The 
Pearson product-moment correlations between 
the samples in the odd/even split for the 11 
subordinate structures were far belbw accept- 
able levels for individual and group measure- 
ment. The first-half/second-half correlations 
for these 11 measures were even smaller than 
the corresponding correlations from the odd/ 
even split. Thus, a 500-word sample did not 



give a reliable measure of the use of certain 
subordinate structures. Even when consider- 
ing all subordinate structures , the correlation 
between two 500-word samples only reached 
.46. By using the Spearman-Brown prophecy 
formula, it was estimated that two 1000-word 
samples might correlate about .63 for total use 
of subordination, but below .59 for. any meas- 
ure of a particular subordinate structure. The 
1 00 6 -word sample may, then, also be inade- 
quate and too unreliable for measuring specific 
subordinate structures . 

For the synopsis scores, however, the cor- 
relations between samples were higher, and 
for the odd/even pages split they approached 
a useful level. The measures of clause length, 
T-unit length, and sentence length appeared 
to h^ve reached a level of reliability (x > . 69) 
that would be sufficient to obtain reliable 
measures of a group's characteristic writing 
behavior. For individuals, one might still de- 
sire the slightly higher reliability than might 
be expected from 1000-word samples. 



Table 2 

Correlations for 17 Pairs of Writing Measures 
Taken from Two 500-Word Samples 



Syntactical 

Structures 


Frequency 
in 1000 
Words 


Odd/Even 
Pages 
All Ss 


First-Half/ 
Second-Half 
All Ss 


Odd/Even Pages by Groups 


Male 

Ss 


Female 

Ss 


High 

Ability 


Low 

Ability 


Adjective Clauses 


10.49 


.42 


.35 


. 53 


.31 


.34 


.52 


Adverb Clauses 


17.73 


.41 


.31 


.46 


.36 


.52 


.27 


Noun Clauses 


14.21 


. 26 


.08 


.24 


. 27 


.45 


.02 


Clauses of Comparison 


.93 


-.01 


-.12 


.05 


-.07 


.01 


-.03 


Deferred Subject 


.16 


.01 


.03 


. 25 


-.06 


-.08 


■. 20 


Cleft Sentence 


. 20 


.29 


.11 


.46 


.08 


. 23 


.35 


"The More, the Merrier" 


.03 


-.02 


-.01 


-.03 


--- 


— — 




Subjunctive 


.01 


— 


-.01 


— 


-- 


— — 


— — 


Special Which Clauses 


.19 


.08 


-.05 


-.09 


.30 


-.09 


. 22 


Adjective Complement 


.24 


-.04 


.10 


-.05 


-.07 


-.04 


-.07 


Adverb Complement 


.13 


.04 


.07 


.19 


-.06 


.13 


-.06 


All Subordinate Clauses 


44,33 


.46 


.39 


.49 


.43 


.51 


.39 


Synopsis Scores 


Mean 




Clause Length 


, 8.32 


.69 


.48 


.72 


.64 


.72 


.64 


Clauses per T-unit 


1.59 


.54 


.22 


.56 


.52 


.65 


.42 


T-unit Length 


13.22 


.74 


.56 


.81 


.68 


.82 


.61 


T-units per Sentence 


1.25 


.65 


.48 


.72 


.51 


.69 


.62 


Sentence Length 


16.47 


.77 


.62 


.76 


.79 


.79 


.75 
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In comparing the two methods of sampling 
from a writing sample, odd/even pages vs. 
first-half/second-half, it is obvious that the 
odd/even pages procedure was superior. For 
the synopsis scores the odd/even correlations 
ranged from .54 to .77 compared to a range of 
, ' 7 to .62 for the first-half/second-half . That 
iSj sampling every other page represents the 
entire corpus of Writing more adequately than 
if one considers only the first half. Perhaps 
the ideal procedure would be to consider every 
other sentence, or every third sentence, or 
every fourth sentence, etc., depending on the 
proportion of the total corpus one wanted to 
sample. 

In Table 2 the correlations between writing 
samples taken from the odd/even split are pre- 
sented for male, female, high IQ, and low IQ 
Ss separately. Several differences between 
pairs of correlations are apparent, but what 
these differences indicate about writing per- 
formance is uncertain. Possibly the writing 
of the male and high IQ ^s was more stable 
and consistent since the correlations for these 
two groups were generally higher than for fe- 
male and low IQ £s . Consistency does npt, 
however, imply maturity. 

In Table 3 the intercorrelations between the 
odd/even pages synopsis scores are presented. 
All correlations greater than . 26 in absolute 
value show significant relationship (£< .01) 
for 133 degrees of freedom. Note that measures 
of clause length, clauses per T-unit, and T- 
units per sentence are largely independent 



measures of writing behavior. That is, a per- 
son who writes long clauses does not neces- 
sarily write more or less clauses than someone 
who writes short clauses. If clauses per T- 
unit is considered a measure of subordination, 
and T-units per sentence a measure of coor- 
dination, it is apparent from Table 3 that at 
Eighth Grade these two skills are uncorrelated. 
Clause length, sentence length, and T-unit 
length, however, are significantly correlated 
since sentences are made up of T-units and 
clauses, and T-units include any clauses 
present. 

The following conclusions become apparent 
from the comparison of two 500-word samples: 

1. Several structures (clauses of compari- 
son, deferred subject, cleft sentence, 
"the more, the merrier," subjunctive, 
special which clauses, adjective com- 
plement, and adverb complement) were 
very infrequent in 500 words and also 
1000 words. Much larger samples of 
writing covering various writing situa- 
tions would be needed to make reliable 
estimates of their incidence. 

2. Adjective, adverb, and noun clauses 
together occurred about 5, 9, and 7 
times, respectively, in an average 500- 
word sample of Eighth-Grade writing. 
The frequencies correlated from . 26 to 
.42 from sample to sample (odd/even 
pages) . These correlations seem too 
low for most purposes, and even the 



Table 3 

Intercorrelations of the Five Synopsis Scores 
in the Odd/Even Comparison* 



Variable 



Odd Pages Even Pages 

W/C C/T W/T T/S W/S W/C C/l W/T T/S W/S 



Odd Pages 



Clause Length 


(W/C) 


1.00 -.03 .77 


Clauses per T-unit 


(C/I) 


1.00 .60 


T-unit Length 


(W/T) 


1.00 


T-units per Sentence 


(T/S) 




Sentence Length 


(W/S) 




Even Pages 


Clause Length 


(W/C) 




Clauses per T-unit 


(C/T) 




T-unit Length 


(W/T) 




T-units per Sentence 


(T/S) 




Sentence Length 


(W/S) 





-.22 


CO 

LO 


.69 


.12 


cn 

00 


-.28 


.32 


LO 

o 

1 


.51 


O 

CD 


.54 


.45 


.03 


.44 


-.21 


.74 


.59 , 


.45 


.74 


-.20 


.54 


1.00 


.49 


-.01 


-.06 


-.04 


.65 


.42 




1.00 


.53 


.34 


.63 


.27 


.77 



i—‘ 

O 

O 

1 

o 

cn 


.67 


I 

i—‘ 

cn 


.52 


1.00 


.70 


-.24 


.47 




1.00 


-.28 


.72 






1.00 


.45 








1.00 



* r > .26 is significant at the ,01 level 






higher: estimatea from a larger sample 
(1000-words might correlate .41 to .59) 
would be inadequate for measures of 
individual performance . 

3. The total number of all subordinate 
structures reached a sample-to-sample 
reliability of .46, also too unreliable 
for most purposes. 

4. The synopsis scores showed relatively 
high levels of reliability compared to 
the subordinate structure frequencies. 

5 . Though groups of S.s did not differ sig- 
nificantly from sample to sample, dif- 
ferent sampling procedures varied in 
how well they represented a larger sam- 
ple. In the present case, comparing 
odd to even pages resulted in higher 
correlations than comparing first-half 
to second-half samples. 

Thus, it appears that at least three of the 
synopsis scores (clause length, clauses per 



T-unit, and T-units per sentence) are non- 
redundant measures that may be obtained ob- 
jectively and reliably from a writing sample of 
500 words. Two other synopsis scores, sen- 
tence length and T-unit length, are also reli- 
able when obtained from 500-word samples, 
but reflect factors which are already measured 
by the other three synopsis scores. Sentence 
length and T-unit length are therefore corre- 
lated more highly with the other synopsis 
scores . 

The reliability of the synopsis scores from 
500-word samples appears to compare favor- 
ably to the levels usually obtained in the 
measurement of wilting. Apparently, sampling 
systematically (every other page or sentence) 
will provide a reliable representation of the 
larger corpus of writing, as far as the synopsis 
scores are concerned. For measures of the 
frequency of specific types of clauses, the 
only option appears to be the tabulation of 
very large writing samples over many types of 
topics and situations. 
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